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ABSTRACT 



This publication documents a participatory assessment 
project conducted from 1993 to 1995 with four afterschool programs in New 
York City. Key questions were whether the assessment project would develop 
the capacity of program staff to assess their own projects and whether the 
programs were successful in supporting the literacy development of the young 
people they served. Also considered were aspects of the programs positively 
associated with literacy development and lessons about the effectiveness of 
participatory assessment. Chapter 1 discusses the conceptual framework of the 
participatory assessment process and how it was implemented in this study. 
Chapter 2 describes the philosophical framework underlying the four programs 
and provides brief project profiles. These programs were characterized by a 
focus on young people as resources, not problems, and on literacy as 
involving many different practices fostered by active learning approaches. In 
chapter 3, critical aspects of the assessment process are described, and 
chapter 4 presents recommendations for participatory assessment projects and 
for strengthening afterschool and other youth- serving programs that support 
literacy among young people. The assessment process was effective in helping 
programs clarify their objectives and evaluate themselves. It also 
contributed to an understanding of the factors that make afterschool literacy 
programs successful, including a philosophy of youth and literacy development 
and a pedagogy that emphasizes teaching for meaning and active participation 
of the learner in an environment that is not school -like. (Contains 22 
references.) (SLD) 
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Black children in the cotton fields 
Black man in chains 
Black women suffering in silence 
Watching their men and children in pain. 

Our people are tired of being oppressed. 

Generations of our people suffered never to rest. 

Now our children are dying and mothers are crying. 

In the face of violence we can't keep our silence. 

When will we get to rest? 

(Written by participant of the Challenge Program , 
Interfaith Neighbors) 
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Introduction 

his publication documents a participatory 
assessment project, conducted from 1993 to 
1995 with four afterschool programs in New York 
City. The project grew out of a 1991-92 initiative 
funded by the New York City Department of Youth 
Services in which the Academy for Educational 
Development (AED), a nonprofit educational 
evaluation and technical assistance organization, 
with offices in Washington, D. C. and New York 
City, provided technical assistance to youth-serving 
agencies. The main goal of this technical assistance 
was to increase program staffs capacity to assess their 
programs for purposes of planning, program 
improvement, and communication with key 
constituencies— parents, community and board 
members, and funders. AED learned a great deal 
from the project about the power of a participatory 
assessment project to promote learning within 
organizations and to build the capacity of program 
directors and key staff in youth-serving agencies to 
become more reflective about programming and its 
impact on young people. 

In 1993 the Robert Bowne and Charles Hayden 
foundations joined the Department of Youth Services 
in funding a project that would draw on what had 
been learned in the earlier initiative but focus 
exclusively on afterschool programs that 
incorporated innovative approaches to literacy 
development. The goal of this project was to conduct 



assessments of each program in collaboration with 
key staff in order to answer the questions presented 
below. 

We address these questions throughout this 
report, which is organized as follows: 

• Chapter one discusses the conceptual 
framework of the participatory assessment 
process, its objectives, how it differs from 
other types of participatory research, and 
how it was implemented in this project; it 
also describes project activities and 
summarizes the lessons learned from the 
project. 

• Chapter two describes the philosophical 
framework underlying the four participating 
programs and provides brief project profiles. 

• Chapter three discusses critical aspects of the 
assessment process; findings about program 
effectiveness in promoting the literacy 
development of participants; and lessons 
learned about the benefits to programs of 
participating in the assessment. 

• Chapter four presents recommendations for 
participatory assessment projects and for 
strengthening afterschool and other youth- 
serving programs to support literacy among 
young people. 




— 

KEY ASSESSMENT QUESTIONS 

• Would the assessment project develop the capacity of program directors and key staff to assess their own 
programs? How would they use what they learned from the project in future planning and program 
development? 

• How successful are the programs currently in supporting the literacy development of the young people whom 
they serve? 

• Which aspects of the programs are positively associated with the literacy development of participants? 

• What lessons could be learned about the effectiveness of participatory assessment in building staff capacity for 
internal assessment and reflection that could be applied more broadly to bring about improvement in 
afterschool programming? 
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Chapter One 

Conceptual Framework and Project Activities 

Conceptual Framework 

I n conceptualizing this project, AED was 
guided by a need expressed in discussions 
with program staff from youth-serving agencies 
around the country. Although they felt that they 
played a vital and positive role in supporting the 
development of young people, program staff had 
little systematic data to prove this. In a decade of 
increasing demands for public accountability for 
dollars spent on services to youth and of decreasing 
funding for youth programming, program staff felt a 
great deal of pressure to demonstrate their 
effectiveness to the general public and funders. In 
addition, as reflective practitioners, they wanted to 
know how well they were carrying out what they 
intended to do; how they could strengthen their 
programs; and how they could communicate their 
"successes" more effectively to community members, 
boards, and funders. 

With these considerations in mind, AED 
developed a project that would involve key program 
staff in assessing their own programs and would 
leave them with analytic tools and documentation 
instruments for ongoing reflection and assessment. 
With foundation support, AED also planned to 
conduct an assessment of participating afterschool 
programs that would be technically rigorous and 
would yield useful information to programs and 
funders and others interested in understanding the 
impact of programs on young people's development, 
especially their literacy development. In order to 
accomplish these objectives, the project had four 
components, summarized in the next box. 

Participatory assessment seeks to "maximize the 
usefulness of the evaluation data for intended users" 
(Cousins, 1995). In this respect, participatory 
assessment differs from other participatory 
approaches to research, which have as their main 
goal the redefining of power relationships or 
addressing social inequities through the participation 
of traditionally underrepresented members of a 
community— for example parents in schools— in 
research projects on issues affecting their lives 



ASSESSMENT PROJECT COMPONENTS 

• Identifying programs with the staff capacity and 
commitment to participate in a two-year, labor- 
intensive project and the organizational support to 
do so, including a commitment to use the assessment 
results to guide program planning and development 

• Training key staff in assessment approaches and 
methods 

• Providing technical assistance to the programs in 
developing assessment questions and an assessment 
design for carrying this out 

• Analyzing/reviewing findings and reports or any 
published materials about the project 



(Weiss, 1992). While these goals may be aspects of 
the participatory assessment process described 
here— and we will return to them in the discussion of 
the assessment process in each program — the 
essential goal of participatory assessment, as we use 
the term, is to be of use to program staff in their 
ongoing work and future planning, as well as to help 
them be accountable to their communities, boards, 
and funders. Unlike stakeholder models of 
assessment in which large numbers of stakeholders 
are involved in the development of research 
questions and in the review of findings, the model of 
participatory research used in this project limited the 
number of stakeholders to two or three per program 
(usually a program director and a senior staff person 
or, in some cases, an agency director and program 
area director) who were likely to use the assessment 
approaches and findings. In contrast to stakeholder 
research, the participatory model requires program 
staff to acquire some of the technical skills of 
evaluation as well as to participate intensively in data 
collection and analysis— that is, the participants and 
evaluators mutually determine and control all phases 
of the assessment. For this to happen, participants 
must devote a considerable amount of time both to 
learning evaluation approaches and strategies and to 
carrying out evaluation activities in their programs 
(Cousins, Earl 1992). The table on the next page 
summarizes thfese three approaches to assessment. 
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There were many unexpected outcomes of the 
assessment process, but perhaps the most notable 
one, and one that changed the character of the 
project, was the amount of networking and mutual 
support that occurred among participating programs. 
Originally designed to assist each program in 
carrying out its own assessment, the project in fact 
helped each program understand other afterschool 
programs in some depth; develop sufficient trust 
among themselves to discuss difficult issues of 
programming; and share both assessment and 
program strategies. Sharing of assessment instruments 
and program ideas became common practice soon 
after the project began. Developing this community 



of reflective practitioners across programs was one of 
the unexpected and most important outcomes of the 
project. We will return to this aspect in the 
discussion of the project's impact on the programs in 
chapter three. 

Project Activities 

The rest of this chapter describes each of the four 
components of the project. 1) identifying programs 
for participation; 2) training key staff in assessment 
approaches and strategies; 3) providing technical 
assistance to programs; and 4) analyzing/reviewing 
findings and all published materials on the project. 
The lessons learned from the project are summarized 
at the end of the chapter. 



APPROACHES TO ASSESSMENT 
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. Desired Outcomes l||lfl 


Participatory assessment 


Staff in leadership 
positions 

Other staff/parents and 
participants 

Technical assistance/ 
evaluation experts 


Participation in all 
aspects 

Discussion of methods 
and findings 


Staff, especially in lea- 
dership positions learn 
to assess outcomes of 
their programs and to 
use outcomes in 
planning, program 
improvement, 
communication 


Participatory action 
research 


Underrepresented 
groups such as parents, 
adult literacy students 

Technical assistance/ 
evaluation experts 


Participation in all 
aspects of research 


Represent multiple, 
formerly underrepre- 
sented voices in 
assessment of social 
change projects in order 
to bring about 
improvements in their 
lives 


Stakeholders evaluation 


Key stakeholders in large 
projects 

Technical assistance/ 
evaluation experts 


Participate in defining 
evaluation questions and 
reviewing findings 

All aspects of evaluation 


Develop ownership of 
key stakeholders in the 
evaluation and its results 
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1. Identifying programs for participation 

Not all programs are prepared to engage in 
participatory assessment. Having already conducted 
one pilot project with other youth-serving agencies in 
the city, AED was aware of some of the 
preconditions for successful participation in such a 
project and had already developed a selection 
process to ensure that participating programs were 
prepared for the work and would be able to use the 
results. The process developed for this project 
entailed developing a Request for Proposals to be 
sent to selected afterschool programs that received 
funding from the New York City Department of 
Youth Services and The Bowne and Hayden 
foundations. Following this, AED held a bidders' 
meeting in which staff outlined project goals and 
invited programs to discuss the types of questions 
they were interested in pursuing in the project. In 
listening to program staff discuss questions of interest 
to them, AED staff were able to develop a 
preliminary understanding of the types of issues and 
concerns that program staff attending the meeting 
considered important. Following the bidders' 
conference, programs submitted their proposals; 
these included a description of their organization's 
mission and participants; the questions they hoped to 
pursue in the project; how they proposed to use the 
findings; and which staff would commit their time to 
the project. Based on a review of the proposals, AED 
selected six programs for interviews in which these 
issues were discussed further; four of these were 
selected for the assessment project. 

The four major criteria that AED used in making 
the final decision about which programs would 
participate in the assessment project are listed in the 
next box. 

2. Training key staff in assessment approaches and 
strategies 

The approach to participatory assessment in this 
project is based on a sharing of technical knowledge 
among participants. In order to foster this sharing, 
AED developed a series often two-hour workshops 
that all participating program staff were required to 
attend. The workshops were divided into four parts. 
In the first, AED focused on helping programs 



CRITERIA FOR PARTICIPATION IN THE 
ASSESSMENT PROJECT 

• The commitment of staff to participate in all phases 
of the two-year project 

• The participation of staff in leadership positions in 

which they could use what they learned in the project 
in staff development and to improve their programs 

• Sufficient organizational stability so that staff could 
participate in a two-year project, carry out evaluation 
tasks in addition to their normal work load, and share 
their learning with others 

• Already existing organizational support, as well as 
structures for staff development and inquiry into 
program effectiveness, which could be used to discuss 
the assessment activities and findings 



develop an understanding of different types of 
evaluations and their uses. In the second, program 
staff reviewed their program goals and objectives and 
asked hard questions about how they had been 
developed and whether or not program activities 
supported the attainment of these goals and 
objectives. Part three focused on providing program 
staff with hands-on opportunities to learn about 
selected evaluation strategies, and part four focused 
on assisting program staff to develop the assessment 
questions and an evaluation design. 

The most challenging part of the workshop series 
was the aspect that we at first expected would be the 
easiest— defining program objectives. In many ways, 
the agencies had limited the description of their 
programs' goals and objectives to meet the criteria 
and expectations set by funders and to some extent 
their constituencies. For example, they were 
accustomed to reporting what they did, not why they 
did it. That is, as service organizations, they often 
framed their objectives around the provision of 
services rather than around what they hoped to 
accomplish through service provision. As we 
encouraged programs to define their objectives, a 
further complication became apparent: the various 
program constituencies held somewhat different 
views of program objectives from those held by 
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program leaders or board members. For example, 
although one of the programs provided very 
innovative interdisciplinary, project-based 
approaches in an effort to help students develop and 
improve their literacy skills, the parents— many of 
whom were recent immigrants— were most interested 
in the assistance the program could provide in 
helping their children complete their homework. 

The discussions about objectives led some 
programs to realize that what they had defined as 
objectives were often broad social goals — for 
example, empowering young people to participate 
more fully in their communities. The challenge AED 
posed to program staff was to be precise about the 
skills, knowledge, beliefs, attitudes, and behaviors 
they wanted young people to develop in their 
programs in order to accomplish these broad social 
goals. We encouraged staff to think about how they 
could determine whether or not they were 
accomplishing their objectives and whether they and 
their program activities supported young people in 
developing these skills, knowledge, attitudes, beliefs, 
and behaviors. Much of the learning about 
assessment and the conceptual framework needed to 
carry out assessments emerged from the discussion of 
program objectives. 

The workshops series presented materials on the 
various strategies used in assessment— observation, 
interviews, focus groups, and surveys. In addition, 
AED staff discussed and developed forms to 
document which young people the program served 
and the types and intensity of services. All the 
workshops were participatory: program staff had 
opportunities to learn about assessment strategies, 
review assessment instruments, and develop and test 
their own instruments. For example, following a 
discussion of surveys, participants reviewed and 
critiqued various types of surveys used in assessment 
and, as a group, designed their own parent survey. 
Several programs pilot-tested the survey with a group 
of parents and reported the results at a subsequent 
workshop. Further refinements were then made in 
the original survey instruments based on the results 
of the pilot. 

The final phase of the workshop series involved 
the development of an assessment plan that specified 



the questions each program wanted to address; a 
description of the methods staff would use to answer 
the questions and what staff would carry out the 
assessment activities; and what products staff wished 
to have from the evaluation— for example, forms to 
be used for internal program documentation and data 
to report to their boards or funders. 

An additional task of the project was for program 
staff to present their assessment questions and 
designs to other afterschool programs in the city. 
This was carried out in a four-hour workshop, the 
main objectives of which were to help participating 
programs clarify their designs enough to present 
them to external audiences and to inform other 
programs about the project; provide other programs 
with instruments they could use immediately to 
document their program activities and outcomes; and 
convince other programs that the process was useful 
and achievable, and to interest them in participating 
in a similar assessment process. 

3. Providing technical assistance to programs 

Programs would not have been able to develop 
an assessment design or carry out the activities 
without a great deal of technical assistance from AED 
staff. As one staff member put it, "It was like pushing 
a stick through molasses; I was just not used to 
thinking this way.' Interesting insights into 
assessment emerged from contrasting how program 
professionals and assessment professionals think. 
Program staff have been trained in their professions— 
as teachers and social workers or youth workers — 
and think programmatically: What are the needs of 
young people? How does my professional 
knowledge help me to meet them? How can I meet 
them within the constraints of my budget, agency, 
and staff capacity? AED asked staff to think about 
other issues: What are the conceptual underpinnings 
of the program? On what theory or theories is it 
based? Are these underpinnings consistent or 
contradictory? How are they reflected in the program 
objectives, activities, and organizational structure? 
How do the staff and other constituents, such as 
parents, conceptualize the program objectives? How 
do staff, students, and parents determine program 
effectiveness? How do staff know how effective they 
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are with young people? How can staff document 
change over time? How can they use what they find 
out to change or modify their program and present 
the findings to others? 

The technical assistance provided was also very 
practical in nature. AED staff helped programs 
develop new intake forms to obtain the information 
they needed about participants and to develop 
instruments for measuring program outcomes. In 
addition, we modeled some evaluation strategies that 
program staff wished to use. For example, we 
modelled a classroom observation, helping 
participants address relevant questions: In the welter 
of impressions from a classroom, what do you write 
down and why? What is the nature of interactions 
between teachers and students and students and 



students in the classroom? How do you summarize 

and analyze your notes? In addition, we helped pilot- 

test instruments— surveys of young people and their 

parents— and analyzed the results with staff. AED 

also carried out some activities — for example, focus 

groups with students touching on sensitive issues that ’ 

might be critical of the program design or individual 

staff members— that program staff felt would be more 

effectively carried out by staff from an outside 

organization. Finally, we talked about products— 

charts, quotes, "success stories"— that could be § 

presented to various audiences regarding program 

accomplishments. 

AED collaborated in conducting the assessment 
itself. Our tasks were two-fold: collecting school data 
for all students and developing a case study of each 
program. These tasks are described below. ® 





ASSESSMENT TASKS 




Collecting School Data for All Students 


Goal: 


To document the impact of the programs on school achievement. 


Method: 


AED collected data from program staff who had obtained some of it during the intake 
process. Program staff also obtained this information from the schools and individual 
teachers. 




Developing a Case Study of Each Progrcun 


Goal: 


To develop a case study of each program that would provide a rich contextual 
description of that program’s approaches and activities; the staffs backgrounds and 
perceptions of their roles and responsibilities; parent perceptions of the program’s 
value for their child; and young people’s perceptions of program strengths and 
weaknesses. 


Method: 


Week-long observations, interviews, and focus groups; selected observations 
throughout a year 
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4. Analyzing/reviewing findings and reports 

As data were collected, AED met with program 
staff to analyze the findings. In the case of surveys, 
AED developed the analysis plan and entered and 
analyzed the data. We discussed the significance of 
findings with staff and, in the case of pilot surveys, 
refined the instruments so that they would yield 
more useful data. For example, a parent survey that 
asked parents how their child might act in various 
situations in the program was difficult for parents to 
answer. They preferred open-ended questions in 
which they could reflect on their child's behavior at 
home. As a result, the parent survey was 
considerably revised to combine open- and close- 



ended questions. The case studies of each program, 
which AED developed, were sent to the programs for 
review, and in a meeting of all program staff, there 
was a discussion of the final report for the project. 
Each program commented on the case-study's 
accuracy in reflecting the program and its 
accomplishments, as well as on the impact of the 
project on the program. Program staff also made 
recommendations about the content of this 
publication— that is, which aspects of the project 
were most important to highlight for other audiences. 

The phases of the assessment, for both AED and 
programs are illustrated in the box below. 



PHASES OF THE ASSESSMENT 





'it l-'-X Ilf. Programs 


Phase I 


Phase I 


• Present hands-on workshops ( 1 0 weeks) 


• Learn evaluation strategies and thinking 


• Facilitate plan design 


• Define goals and objectives 


• Record and analyze the development process 


• Develop assessment plan 

• Present plan to other programs 


Phase II 


Phase II 


• Assist with assessment 

• Conduct some assessment tasks 


• Conduct the planned assessment 


• Assist with or conduct analyses and 


• Discuss methods and results with other staff 


summarize results 


and parents 


• Summarize (each) program's experience in 


• Review on AED final report 


phases I and EL including results of their 
assessments, in four case studies about the 
participating programs 




• Produce a final report 

• Incorporate programs’ changes and 




suggestions 
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Lessons Learned from Assessment Project 

1 . For participating program staff, the 
experience of carrying out an assessment of their 
programs was difficult and also powerful. Carrying 
out the assessment was difficult because it was 
extremely time consuming, it demanded new ways 
of thinking, and the results were sometimes difficult 
to take— evidence of program weaknesses or staff 
criticisms were disturbing. As one staff member said, 
"Others should not take on this project unless they 
are prepared to hear and act on the findings." 
However, because staff believed the assessments 
were valid, the information provided was used to 
guide change and strengthen the programs. In 
addition, staff felt empowered by the experience 
because it gave them a new way to look at their 
programs. Indeed, as described above, thinking as an 
evaluator involves a paradigm shift for program staff: 
it involves moving back from their programs and 
observing them— ultimately a powerful experience 
since it provided new tools of analysis, new ways to 
look at what they were doing, and new ways to 
approach program development. 

2. The project is very labor intensive for 
everyone involved. The project took much more 
time than originally anticipated or budgeted. In 
addition, some programs suffered cutbacks in 
funding during the project, and staff in these 
programs had less time to spend on the project and 
had to eliminate some aspects of their assessment 
design. One program director noted that only 
full-time staff could participate since part-time staff 
were not paid for extra meetings or other types of 
activities. Further, there were some gaps in 
communication between AED and program staff 
during the final stages of data analysis and writing of 
the final report because of the pressure of deadlines. 

3. Organizational support for participation in 
such a project is essential. In all cases, the agency 
fully supported the participation of the staff in the 
project (in two cases the directors of the agency 
participated). Such a level of organizational support 
is probably unusual, particularly in large agencies, 



but essential for sustaining work in the project over 
time, for communicating with the rest of the staff, and 
for applying the findings to program planning and 
improvement. 

4. Having an external, trained evaluation team 
is essential to the success of a participatory 
assessment project. In this project, the team provided 
external validation of assessment designs, data 
management review, and the technical capacity to 
enter and analyze data. Working closely with an 
assessment team helped to demystify the process and 
build the capacity of the programs to broker services 
from evaluators in the future. 



* We use the term "assessment" because of its recent 
association with approaches to student assessment 
emphasizing the relationship between assessment and 
instruction and hence its formative character for teachers and 
students; the participation of students in their own 
assessment; and the goal of having students become 
proficient in determining the strengths and weaknesses of 
their own performance through discussion with teachers, 
peers and other reviewers of their work. All these aspects of 
assessment were important in the project described in this 
report. 
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Chapter Two 

Program Philosophy and Profiles 
Program Philosophy 

T his chapter discusses the philosophy that 
guided the development of the four 
afterschool programs. Because this project focused 
on programs known for innovative approaches to 
literacy development, it was important to understand 
their perspectives on literacy and youth 
development. AED developed this understanding 
throughout workshops and through the interviews 
and observations that we conducted for the case 
studies. These perspectives were very similar and led 
to similar types of activities in spite of such obvious 
differences as the size of the sponsoring agencies, the 
young people whom they targeted, their program 
operation (especially schedules), and staff 
backgrounds and staff/ participant ratios. The 
discussion of program perspectives on literacy and 
youth development is followed by a brief description 
of the four programs illustrating how they applied 
these philosophical perspectives in their 
programming. 

The following three perspectives on youth 
development and literacy characterized all the 
programs in the project: 

1. A perspective on young people that views 
them as resources and not as problems in need of 
"fixing." Each program targeted students from 
among the poorest communities in the city — 
communities characterized by high poverty and 
unemployment rates and other indicators of poverty, 
including high rates of teen parenthood and infant 
mortality. In the two programs for which data were 
available, 38 and 88 percent of the participants were 
eligible for free or reduced-price lunch. Two 
programs specifically targeted young people who 
were behind in school as measured by their scores 
on standardized tests in reading and mathematics. 
Many program participants were in special education 
or bilingual programs. Two programs did not 
specifically target students with academic difficulties; 
nevertheless, 48 and 44 percent of their participants 
were below grade level in reading, and 56 percent 



and 33 percent were below grade level in math. 

In spite of the fact that each program recruited 
students commonly labeled as students with 
academic deficits in need of remediation, program 
staff adopted a different perspective on the young 
people in their programs. They believed — and young 
people's stories confirmed their view— that many 
young people had difficulty in school because 
teachers were not able to develop relationships with 
them that allowed them to learn at their own speed 
and in their own fashion. One director said of his 
participants that they had learned early on that the 
best way to get by in school was to go unnoticed. 
Many of his students barely expressed themselves at 
all when they first entered the program, preferring 
invisibility to the possible ridicule of peers. In other 
programs, many participants also had histories of 
poor achievement and others of withdrawn or acting- 
out behavior in school. 

All the programs believed that all young people 
—whatever their academic performance— have 
voices, feelings, and opinions waiting to be tapped 
and articulated, and defined their major task as 
helping young people tap their dreams, articulate 
their thoughts, and imagine their futures. An essential 
aspect of strengthening young people's potential in 
each program included providing them with 
opportunities to understand their social realities and 
to identify and celebrate the strengths of their families 
and the communities to which they looked for 
inspiration, role models, and guidance. This aspect 
also included providing young people with a voice 
in deciding the content and direction of the program. 
Such perspectives are grounded in recent research in 
what makes youth programs effective. Of critical 
importance is the perspective on young people as 
resources to be developed rather than as problems in 
need of fixing (Cahill and Pittman, 1992; 
McLaughlin, etal, 1994). 

2. A perspective on literacy that emphasizes the 
many types of literacy practices and purposes and 
the need for developing supportive communities in 
which to develop literacy skills. All the programs 
viewed literacy as involving many different practices, 
such as completing a worksheet in a class; assisting 
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a non-English speaking parent to buy food or 
negotiate a social service agency; and listening to, 
understanding, and retelling a story told by a beloved 
grandmother. In other words, programs viewed 
literacy as practices entailing many different types of 
skills used in different settings— families, schools, 
communities and institutional settings— for many 
different purposes. School literacy skills are among 
the many that young people may practice. 

Sylvia Scribner has described literacy practices 
and purposes using three metaphors: "literacy as 
adaptation," "literacy as power," and "literacy as a 
state of grace" (Scribner, 1988). The first metaphor, 
literacy as adaptation, describes literacy in everyday 
usage— reading directions or filling out forms. The 
second metaphor, literacy as power, points to the 
power differences between those who are the most 
and least competent in the literacy practices that are 
valued in a given society and that are signs of and 
reinforce the distance between social groups. 
According to the Brazilian educator Paolo Freire, the 
most important task of literacy development among 
poor people is its use in social transformation. As 
people learn to name the social realities around them 
by reading and writing, they also acquire the tools to 
change them. The third metaphor, literacy as a state 



of grace, refers to literacy acquired for personal 
edification and satisfaction through an individual 
partaking of the intellectual, artistic, and spiritual 
legacy of all cultures and the status conferred on 
those who acquire this type of literacy. In the 
programs in this project, all these forms of literacy 
were deemed important: the literacy necessary for 
living in today's society, the literacy required to 
name and change one's social realities, and the 
literacy needed to become knowledgeable about and 
to enjoy the world's artistic, philosophic, and 
spiritual traditions. 

Participating programs emphasized the relational/ 
social aspects of literacy as particularly important. In 
this view, literacy practices occur in social settings as 
part of the process of communicating with others— in 
a church when the Bible is being read, at home 
when notes are left for family members, in school 
when instructions for an in-class experiment are read 
by a group of students— to name but a few settings. 
A thread running through all the programs was 
attention focused on developing communities among 
adults and children characterized by trust, support 
and respect— something that many participants 
lacked in their school settings— in which literacy 
practices could flourish. 
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You need to have a reason to read and write. Being part of a community in which what 
you read and write is shared and valued is one of the main reasons for developing literacy 
skills. We begin in our program by developing a sense of community. 

[Director of afterschool program participating in assessment project] 

I n the program young people have to open themselves up to varying points of view. 
Students are not passive here. They are more competent in espousing their likes and 
dislikes about things. They feel involved enough to make a decision. 

[Staff person in afterschool program participating in assessment project] 

We want our participants to see themselves as individuals who take action, rather than as 
individuals who have things done to them. 

[Staff person in afterschool program participating in assessment project] 



3. A perspective on teaching and learning that 
emphasizes active learning approaches within the 
program as important contributors to young 
people's development and ability to become better 
readers, writers, problem solvers, and thinkers. In 
contrast to many traditional classrooms, where there 
is little choice about what is learned, in which rote 
learning rather than understanding is emphasized, 
and in which skills are often taught for their own 
sake rather than in the context of meaningful 
activities or content, all participating programs were 
committed to the following practices: 

• Giving young people choices about what 
they learned and about the activities they 
participated in so that they could choose 
topics and issues of importance and meaning 
to them 

• Teaching skills— such as spelling or punc- 
tuation-through meaningful and engaging 
projects and activities 



• Helping young people make meaning out of 
their lives through literacy activities — for 
example, writing activities that helped them 
identify what they valued, learn about their 
backgrounds, clarify their goals, and name 
and reflect on such social realities in their 
lives as violence in their communities or 
difficulties dealing with peers in school 

There is a growing body of research that supports 
the effectiveness of instruction that encourages active 
learning. Through active learning, students 
participate in the construction of meaning— also 
referred to as a "constructivist" approach to learning. 
Recent research on active/constructivist instruction 
has demonstrated its effectiveness in developing 
higher level thinking skills and overall academic 
achievement. A study for the U.S. Department of 
Education, conducted in 140 classrooms in 15 
elementary schools across the country with high 
concentrations of low-income students, found that in 
those classrooms in which teaching focused on 
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"meaning and understanding," including teaching 
skills in meaningful contexts (that is, never apart from 
engaging and relevant content) and teaching for 
comprehension and meaning, students performed 
better on tests of advanced skills than in classrooms 
without such instructional practices; this was true 
even when taking into account initial differences in 
achievement and poverty level (Knapp, Shields, 
Turnbull, 1992). Additional support for active 
instructional approaches supporting the construction 
of knowledge comes from cognitive psychologists, 
who emphasize the importance of school learning 



that mimics how people learn in real-life settings, 
including workplaces (Resnick, 1987). Others have 
demonstrated positive achievement results when 
teachers carefully scaffold learning experiences to 
assist students in making meaning out of texts. Their 
ideas of scaffolding are drawn from the workplace— 
the process by which expert workers teach novices 
complex new skills (Collins, Brown et al ., 1 991). 

The following program profiles provide evidence 
of the ways in which the programs in this project put 
these ideas about literacy and active learning into 
practice. 
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The Friendly Place/El Sitio Simpatico 
Afterschool Program 

The Agency: The Friendly Place/El Sitio Simpatico— was a community center located in East Harlem New York/ 
a community composed primarily of African-American and Latino/a residents, with many of the latter 
Spanish-speaking immigrants. The heart of Friendly Place was its library, which opened to the street with floor-to- 
ceiling windows that invited passersby to look in; its entrance door had posters of famous people of diverse 
backgrounds with messages fostering literacy. The book-lined shelves were arranged by topic and included 
multicultural books and books in Spanish and English. The center targeted its services to local 
residents — primarily from nearby housing projects — who were encouraged to borrow books and send their 
children to the afterschool program. 

The Program: The afterschool program targeted children in grades 2-6. During the 1995-96 program year, 39 
students were enrolled— almost evenly split between children of African-American and Latino/a backgrounds. 
Staff indicated that many children entered the program behind in their reading and other academic skills. Staff 
consisted of an education director and two full-time and one part-time employee who worked with students in 
groups of approximately 12. During the assessment project, the program’s funding was cut, and the program lost 
one of its full-time staff members. The program director assumed the responsibility of a group leader to replace 
this person. 

According to the center brochure, the mission of the afterschool program was to: 

Improve the literacy and critical thinking skills of young people by providing a safe and 
nurturing environment built on mutual trust and respect among participants and adults. . . . We 
want children to have fun while they learn and develop a sense of ownership of the program 
and the work they do here. We use a whole-language approach, believing we must address 
the whole child, including his/her home, school and community life. 

Program Vignette: Students spent five days a week at center from 3-5:30 p.m. A typical schedule for the day 
included a group meeting, snack time, homework time, book sharing, and theme unit or choice time. During 
homework, students worked together with their group leader; if a child did not have homework that day, an ”1 
ain't got no homework" folder suggested writing or other activities that he/she might pursue instead; all work was 
kept in a folder. Following homework, students frequently read a book from the center's large collection. In this 
they were encouraged to pair with another participant and to write to this person when they had finished a book, 
describing it and saying whether they recommended it and why. Other activities in this time period included a 
theme that the group leaders pursued for a month: African-American culture or women in science were themes 
pursued during the year of the assessment project. Sometimes, the staff identified areas that students needed help 
in from observing them doing their homework and devoted this period of time to mini-lessons in these areas. 

continued 



* This program stopped operation shortly after the project ended because of loss of funding. 
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The Friendly Place/El Sitio Simpatico 
Afterschool Program 

(continued) 



Other days were "choice" days in which students chose from a wide variety of games, crafts, cooking, or other 
planned activities. The approach to literacy embraced both whole-language and direct-skill instruction through 
the mini-lessons and homework assistance. In reading and writing, students were assisted in developing their 
literacy skills through reading and discussing books that they chose and often shared with others. Staff in the 
center were very dedicated to their work and community. Of the many things that children liked about the center, 
the staff was most important; students said they felt cared for and loved. Parents also valued the program for its 
proximity, safety and, above all, its caring staff. 







StaffrStudent 


No. Staff 


No. Students 


Ratio 


3 


39 


1:13 



Schedule 


Structure 


Activities 


3-5:30 p.m. 


Program 


Meeting time 


(Monday- 


Coordinator 


(3-3:15 p.m.) 


Friday) 


Two staff people 


Snack time 
(3:25-3:35 p.m.) 

Homework time 
(3:45-4:30 p.m.) 

Book sharing--theme p choice, 
mini-lesson, free day 
(4:35-5:15 p.m.) 

Clean-up time 
(5:15-5:30 p.m.) 

Dismissal (5:30 p.m.) 
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Interfaith Neighbors 
The Challenge Program 



The Agency: Interfaith Neighbors is located on the Upper East Side of New York. It occupies one floor of New 
York's Psychoanalytic Institute, where the agency has offered a range of programs to young adolescents aged 10 
to 1 3 since 1 954. The programs are designed to help young people stay in school, understand and confront the 
complex and difficult social realities they face, minimize high-risk behavior, and maximize their learning, social 
competence, and healthy development. It targets young people from many public and private schools— some 
local and others in the Bronx, Brooklyn and Queens. Annually, the agency serves approximately 400 young 
people on site and 1 1 00 in schools. 

The Program: During 1994-95, The Challenge Program, which participated in the assessment project, served 28 
adolescents, ranging in age from 11 to 13, three afternoons a week. They were seventh graders from nine public 
and private schools: 52 percent were African Americans, 32 percent were Latino/as, and the rest were white or 
of other backgrounds, and 38 percent of students were eligible for free or reduced-price lunch. The program has 
a strong youth development perspective and uses reading and writing to help young people learn about 
themselves and others, to find their voices, and express their opinions. Through the various program activities, 
young people explore their personal histories; develop a sense of their personal and cultural identity; strengthen 
reading and writing skills; and develop social and personal skills to assist them in navigating specific 
developmental issues and concerns. The literacy approach integrates a whole-language approach to reading and 
writing with general adolescent developmental theory and group work. Staff are primarily experienced social 
workers or interns with social work backgrounds. Five staff work with the Challenge Program in various 
capacities and meet regularly to plan the program and discuss the needs of participants. Staff are dedicated to the 
young people, calling them at home if necessary and seeing them in their schools if there is a special need. Much 
attention is paid in the program to developing students' leadership and decision-making skills and their 
understanding of group norms and processes, and to providing them with barometers to assess and monitor their 
own behavior. 

Program Vignette: The Interfaith "Writing Bill of Rights" is posted on a wall of the agency: Your writing will not 
be graded or marked with a red pen ; You have a right to an audience for your writing if you want one ; You have 
the right to as much time as you need to finish a piece of writing; You have a right not to be made fun of. These 
principles are put into practice in a variety of group activities that take place during the three program days. In 
addition to group meetings designed to support ownership of the program through sharing in planning and 
decision making, young people participate in a curriculum called Path, which explores issues of personal, social 
and cultural identity through reading and writing. On one day, students listened to music of different cultures 
and, working in small groups, brainstormed images evoked by the music. These images included slavery, tribes 
fighting, praying, Caribbean festival, and walking through the rain forest. Then groups wrote poems using these 
images. The poetry written in Path sessions is often displayed on agency walls, and students can present their 
work, if they choose, to their peers. The poem written by one group is presented at the beginning of this report. 

continued 
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Interfaith Neighbors 
The Challenge Program 

(continued) 



Other weekly activities include journal writing, in which students record events of the week and their feelings 
about them, and their case worker writes back to them; and Rap Group, in which boys and girls meet separately 
to discuss topics of concern to young adolescents— relationships, drugs, anger, stress, self-perceptions, transition 
issues, sexuality and so forth. Rap time is a favorite of young people in the program. 

Young people also attend case-management sessions with their counselor twice a month or more, depending 
upon individual needs. Staff described Interfaith as an agency that provides "a safe place to deal with conflict" 
and to learn to respect differences. Parents play an important role in the Challenge Program— from learning about 
the program goals and objectives in the intake process, to attending meetings, potluck suppers and workshops 
designed specifically for them. In addition, a counselor calls each parent twice a month to talk about his/her 
child. 







Staff:Student 


No. Staff 


No. Students 


Ratio 


5 


28 


1:6 



Schedule 


Structure 


Activities 


3-5 p.m .* 


Program Director 


Snack and relax time 
(3-3:30 p.m.) 


3 days a week 


Director of 




(Monday, 


counseling services 


Unit meeting 


Wednesday, 
and Friday) 


Family/schooi 


(3:30-3:45 p.m.) 




liaison worker 


Activity (path curriculum, journal 
writing, rap groups; 3:45-5 p.m.) 




2 social work 
interns 


Library time (once a month) 






Case management (twice a month 
or more as needed) 



* On some days students may come before 3 p.m. and stay after 5 p.m. for case management. 
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Pius XII /North Bronx Family Center 
Afterschool Tutoring Program 



The Agency: Pius Xll/North Bronx Family Center is a multifaceted service agency located in the University 
Heights section of the Bronx. It offers a range of educational, recreational and support services and activities to 
a predominantly African-American and Latino/a population. Located in a three-story house that once served as 
a dormitory of New York University, the property is now leased to the agency by Bronx Community College, 
its current owner. 

The Program: The program that participated in the assessment project is the afterschool tutoring program. During 
the year of the project, the program served 84 students — 40 percent of whom were African American and 60 
percent of whom were Latino/a. The program operates four days a week; students attend the program for two and 
a quarter hours once a week. Each day students are divided by age into three groups— 7-9, 9-11, and 11-13. 
Students are referred to the program by their schools because they are behind in school; many students also 
attend other programs in the agency on days when they are not in the afterschool tutoring program. On any given 
day, there are likely to be about 1 5 students attending, with about five in each group. The program is staffed by 
an education director and three group leaders who are students at Bronx Community College in human services. 
They are assisted by student interns, also from the college. The director closely supervises the program, providing 
the group leaders and interns with opportunities to develop their literacy skills as they learn how to be effective 
group leaders. Students in this program are often very disaffected learners. Therefore, a major goal of the program 
is to reengage them in learning and help them become active learners. The premise behind program activities 
is that students must feel valued members of a community before they will be willing to speak out and share their 
views and feelings. The program is activity-based and seeks to teach literacy skills through engaging students in 
community-building activities that encourage them to express themselves in writing and to seek answers to 
questions that are important and relevant to their lives. 

Program Vignette: On a typical day, one group of students might be learning math concepts through using play 
money; another group might be involved in silent reading followed by a discussion of what they have been 
reading; a third group might be working on a group project based on a theme that has been selected to focus 
reading and inquiry. During the year of the assessment project, the thematic unit was What Survives in the 
Bronx— a theme chosen to help students develop a positive sense of their community by exploring its ecological 
and social environment. The director chose the theme because students had little knowledge or experience of 
their communities and particularly of the natural environment (students debated one day whether or not there 
were pigeons in the Bronx). Students studied worms, wasps, butterflies, coyotes (which had recently been 
identified in the Bronx), plants, and trees; they took walks in their neighborhoods and created maps; and they 
learned about trees and also explored their own roots. When cutbacks occurred in funding for their program, they 
talked about them in terms of their own survival and wrote letters to the mayor protesting the cuts. 



continued 
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Pius XII /North Bronx Family Center 
Afterschool Tutoring Program 

(continued) 



Typical of this program was the culminating activity of the ecology unit. Students planned and organized a 
museum on What Survives in the Bronx: student work was mounted on the walls, including the neighborhood 
map; posters illustrating how worms and humans differ; and pictures of insects and animals, such as cockroaches 
and coyotes, that survive in the Bronx. Each room had a different focus. In the worm room, students and staff 
performed a worm puppet show, illustrating the usefulness of worms; in the butterfly room, visitors were given 
materials to make their own butterflies; and in the wasp room, visitors learned about the paper-like substance 
out of which wasps build their nests, and students conducted a paper-making demonstration and made 
bookmarks. Throughout this event, students impressed visitors with their knowledge and understanding of the 
issues and their readings and ability to answer questions. This activity is an excellent example of the linkage 
made in this program between building a sense of community and the development of literacy skills in young 
people. 
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Staff:Student 


No. Staff 


No. Students 


Ratio 


4 + tutors 


84 

Daily: 15-20 


1:3 



Schedule 


Structure 


Activities 


3-5:45 p.m. 


Education Director 


Tutoring, writing, library, remedial 
work 


One day a 
week* 


Three group leaders 


(3:30-4:30 p.m.) 




Tutors 


Hands-on activity based on the 
instructional theme 




Three groups of 
students 
(7-9, 9-1 1 , 
11-13) 


(4:30-5:45 p.m.) 



* Program runs Monday through Thursday but different sets of students are seen on each day. 
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Project Reach Youth (PRY) 

Family Learning Center 

The Agency: Project Reach Youth (PRY) is a community-based organization serving the needs of young children 
and adolescents residing in or attending schools in the Brooklyn communities of Park Slope, Sunset Park and Red 
Hook. Located on the outskirts of Park Slope, the agency serves over 3000 youth a year between the ages of 3 
and 19. The majority are from low-income families. 

The Program: For the assessment project, we focused on PRY's Family Learning Center, which provides 
afterschool programs in five schools serving 150 students a year. We focused on two elementary schools with 
a total afterschool enrollment of 105: 70 percent of the students were Latino/a, 1 7 percent African American, and 
the rest of other backgrounds. Most students were in grades 1-4 and were referred to the program because they 
were one grade or more behind in core academic subjects; 88 percent of students were receiving free or reduced- 
price lunch. Students participating in each program attend the school in which it is located. Programs are 
supervised by teachers who also teach in the program. One of the supervisors is also a teacher in the school 
where the program takes place. Both supervisors are responsible for being a liaison with the school staff— the 
principal, guidance counselors, teachers— and with parents. Supervisors also develop the activity-based program, 
which, according to the program literature, seeks "to integrate real-life situations into learning, reading, writing 
and math." 

Program Vignette: On any given day, students can be found first doing their homework in small groups or 
individually for 45 minutes. Following this, students have a wide choice of activities that often involve reading 
and writing around a theme, carrying out projects related to the theme, or participating in a cultural event or trip. 
During the assessment project, the theme was city life. On one day, students were divided into groups and 
assigned the task of drawing a scene of city life. Each scene was cut out and mounted on paper with the other 
scenes to create a collage of the students' images of city life. Before mounting the scenes, each group presented 
theirs to the class, and each group member read a poem he/she had written earlier in the week about living in 
the city. Every student participated and was encouraged by other students and the teacher to present his/her work. 
The project, which is typical of the activities at PRY, offered multiple opportunities to learn and demonstrate 
learning. It emphasized group work and also was successfully completed in a limited time period, leaving 
students and teachers with a feeling of success and accomplishment. Recreation and games are also part of the 
Family Learning Center program; there are often group competitions with a focus on developing students' 
academic skills; cultural events such as story-telling from many cultures are also offered. Croup norms are 
consistently upheld by teachers who call attention to remarks or comments that could hurt children's feelings; 
they discuss the kind of respect that is needed for the program to work for everyone. Parents in the community, 
many of whom do not speak English, are happy that their children are receiving this kind of academic help and 
also that, as one parent put it, "The program also teaches morals. . . how to speak to people, how to be polite." 
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Staff:Student 




No. Staff 


No. Students 


Ratio 


PS 10 


4 


60 


1:15 


PS 124 


3 teachers; 3+ tutors 
(at least one for 
each class) 


45 


1:15 



Schedule 


Structure 


Activities*** 


3-5:15 p.m. 


Two schools* 


Snack 


Two days a 


PS 10** 


Homework 


week* 


Four classes 


* 




(1-2, 3, 4, 5-6) 


Educational activities (literacy 
related, instructional units, cultural 




PS 124 
Three classes 


events, recreational trips) 




(1-2, 3-4, 5-6) 


Read-aloud or choice time 




2 social work 
interns 





* PRY is at more sites; two were selected for inclusion in the study. 

** One small group workshop is held on Friday — topic is selfesteem. 

*** Activities varied by age group: younger students had read-aloud time usually before homework, 
and older students had choice time at the end of the day. 
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Chapter Three 

Assessment Questions, Designs and Findings 

T his chapter discuss how each program 
developed its questions and designs, what 
was found out about each program, and how the 
process of undertaking the assessment affected the 
programs. This discussion is preceded by a summary 
of five aspects of the assessment process that are 
critical to understanding the findings presented in 
this chapter. 

Critical Aspect of the Assessment Process 

First and foremost, the process of conducting 
the assessment was also a program intervention. As 
programs asked questions about the relationship 
between program objectives and activities, they also 
made adjustments in their programs— sometimes 
changing their focus or organization. In our 
discussions of the process, we provide many 
examples of these mid-course changes and also 
summarize the longer term changes that programs 
made and institutionalized. 

Secondly, in discussing the various assessment 
designs, it will be clear that one theme cuts across 
all of the assessment designs — the theme of decision 
making and choice. This theme emerged rather late 
in the workshop series as programs began to share 
their philosophies and program activities in greater 
depth. As AED staff reflected on the assessment 
designs, we hypothesized that the emphasis on 
decision making reflected a theory held by all the 
programs of how participants change and what 
programs needed to do to support change. 

This theory of change suggests that, because of 
the type of schools participants attend and the 
difficult social and economic circumstances of their 
lives, they have few opportunities for meaningful 
choice and decision making. They are rarely asked 
what they are interested in learning; they have little 
choice over how activities are designed in school 
except at recess — one reason school children love 
recess — or whether or not they want to participate in 
them; and they have few opportunities in their lives 



to affect change— for example in their communities. 

Each of the programs in various ways placed 
great importance on providing opportunities for 
informed decision making and choice, and each of 
them closely scrutinized their programs for the 
opportunities that they provided for decision making. 
In addition, the forms developed for the assessment 
asked participants to state their reasons for choices— 
thus the evaluation itself was an intervention that 
further opened opportunities for reflection on choice. 

This theory of change flows from the philosophy 
on which all the programs were based, as described 
in chapter two: namely, a belief that young people 
must be seen as resources and as having valuable 
contributions to make to any program about how it 
should be designed and carried out. In addition, this 
theory reflects the theory of literacy and of 
community building that all the programs espoused, 
which involves active participation of all members in 
building a community within the program — 
including upholding the norms of behavior. This 
theory of change also reflects the perspective on 
literacy as power — the need to involve young people 
in naming their social realities by learning the 
vocabulary to describe them and developing the 
means to communicate what they learn in public 
forums. (The museum event that was the culmination 
of What Survives in the Bronx at Pius XII afterschool 
program — described in the project profile in chapter 
2— was an excellent example of a public forum that 
allowed young people to communicate, in various 
media and formats, what they had learned about 
their communities.) Finally, this theme of change 
reflects an emphasis on active learning, which insists 
that participants be actively engaged in constructing 
meaning. This means that young people must 
understand why they are asked to do something and 
learn to reflect on why they choose to do something: 
Why do they select a particular book to read or 
decide on a particular format for writing or approach 
to solving problems? Becoming more conscious 
about such decisions helps young people to be more 
aware of themselves as learners and hence become 
more active learners. 

Raising the issue of decision making and choice 
was consciousness raising for staff; it demanded that 
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they scrutinize participants' and their own use of 
time and examine program activities for the decision- 
making opportunities they fostered. 

Third, the assessment was carried out during a 
period of tension in the city over budget cuts for 
youth services, which affected programs' ability to 
conduct their assessments. All the programs suffered 
from the cuts, one so severely that it eventually 
closed when the project ended. These cuts also 
undermined the programs' ability to carry out aspects 
of the evaluation. Two programs lost staff, and one 
had to use volunteers instead of paid tutors. All the 
programs had less time for the project because they 
spent a great deal of time protecting their programs 
and advocating against the cuts. Nevertheless, 
programs carried out most aspects of the 
assessment— even under these conditions— an 

extraordinary tribute to their interest in and 
commitment to the project. 

Fourth, the original design for this project called 
for collection and analysis of data from schools on 
participant achievement, which proved difficult to 
execute. The difficulties stemmed from two factors. 
Firstly, the programs drew their participants from 
many schools — up to 22 different schools, both 
public and private— making it difficult to collect data 
and also to have uniform data across schools since 
schools reported tests scores and grades in different 
ways and used different tests. Secondly, programs 
had varying relationships to schools— from no 
relationship at all in one case to being located in the 
schools with direct access to records. However, even 
for the program located in the schools, collecting 
information on grades was extremely onerous. 
Program staff had to sift through all school records to 
identify their participants. The other programs relied 
on participants bringing in report cards, which was 
not consistently carried out. As a result the school 
data is incomplete for all the programs. 

Fifth, as the programs developed their own 
instruments to measure gains in literacy, it became 
evident that the student outcome data from schools 
was not informative about the programs' impact on 



participants' literacy development. Unless the 
program directly focused on improving particular 
school outcomes, it was impossible to determine 
whether or not the program had any impact on 
changes in school achievement. In addition, changes 
in school achievement indicators (e.g. test scores) did 
not necessarily correspond to changes that the 
program documented in participants' literacy. The 
latter were directly related to the program activities 
and were measured by instruments that reflected 
program activities and objectives. Despite these 
problems, we have presented whatever school data 
were collected. They are useful in that they provide 
a baseline picture of participants' school 
achievement and also suggest some of the limitations 
of trying to assess participant progress in afterschool 
programs through the use of school data. 

The Assessment Questions, Designs, and Findings 



Friendly Place 



Friendly Place was in the process of developing 
new directions for its afterschool program at the 
beginning of the project. It had just hired a new 
education director who was breathing fresh life into 
a program that had suffered from a lack of direction 
and considerable staff turnover. Thus the assessment 
was used as a way to plan and strengthen the 
program. Discoveries made along the way resulted in 
program changes or new activities. 

The program developed a series of questions that 
focused on young people's literacy development 
within the program. It drew on the theoretical 
literature on literacy suggesting the multiplicity of 
literacy practices and purposes and asked which 
ones were being developed in the program and 
whether program activities supported literacy 
development in general. 
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Friendly Place 
Assessment Questions 

• What are the types, ranges, and levels of proficiency of literacy 
practices engaged in by children in the after school program? How 
do literacy practices change over time in response to: 

- the activities offered by the program 

- the choices made by the child 

• How do the literacy practices of individual children develop over 
time, as perceived by program staff, parents and children? 

• Does the program design support the program objectives? 

- What types of activities are carried out? 

- How do children perceive the program? 

- How do staff use their time? 

- Does the staff learn form the assessment of children's 
practices and plan accordingly? 



The Assessment Design 

In order to address these questions, staff at 
Friendly Place decided to have participants 
document how they were using their time in the 
program — what were they were actually doing when 
they were supposed to be doing homework and 
what they chose to do in choice time. Staff were also 
asked to document their use of time to see how they 
used it— in planning and supporting participants in 
completing homework and other activities— and 
whether their perceptions and those of the 
participants about the use of time coincided. 

In order to assess participants' proficiency in 
reading, writing and mathematics, staff developed 
instruments to measure many different aspects of 
these skills. For example, in reading, the education 
director used an adaptation of the Primary Language 
Record” to determine whether or not participants 



* The record is a written assessment of a child’s 
development in reading and writing which requires 
ongoing, close observation of children over time. 



were beginning, moderately skilled or fluent readers. 
She sat with individual participants as they read 
books of increasing complexity until they reached 
one they had difficulty with. As they read, 
participants were assessed on their reading strategies 
and comprehension. Originally, the director planned 
to administer the same assessments at the end of the 
program, but, because of the loss of a staff member 
resulting from the budget cuts, she was not able to 
do. 

In order to assess literacy practices and their 
development, staff asked participants to complete a 
log every time they read a book for a week in 
October 1 994 and a week in May 1 995. These logs 
were assessed for evidence of the following: 
participants' being able to state reasons for the type 
of books chosen; participants' sharing their books 
with others by discussing and recommending them; 
and greater complexity in participants' writing about 
the books they read. 
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Findings 

Findings from this program regarding 
participants' literacy development were positive. Staff 
found the following from an analysis of the data 
collected: 

• Use of homework and choice time: 
Approximately 70 percent of participants said 
they had homework and, depending on the day, 
between one-quarter and one-half completed it 
during the program; the majority were given 
worksheets for homework; a high percentage 
were also given writing. During choice time, 
most participants chose to read or play games. 
Teachers' documentation of how they spent time 
corresponded well with participants' description 
of how they spent time. 

• Literacy development: An analysis of 38 reading 
logs from October and May (20 participants 
completed logs in October, 18 in May, and 1 2 in 
both months) showed that the 38 participants for 
whom there were completed logs read 86 books; 
69 percent read these books with other 
participants— that is, they wrote to another 
participant about the book; the rest tended to 
read books alone. 

Participants demonstrated definite preferences in 
their reading, as illustrated in the quotes below. 

"I chose (Chickens' Aren't the Only Ones) 
because it is animal-rific. I like the pictures 
and it tells about chickens and other animals 
that lay eggs." (third-grade boy) 

"I chose (Mighty Babe Ruth) because 
Babe Ruth was one of my favorite 
players and (child's name) told me that it 
was a good book. I liked the book 
because it talks about Babe Ruth and it 
talked about the record he had broken." 
(fourth-grade boy) 



Of the 12 participants who completed logs for 
two months, one-fourth chose more complex 
books as the year progressed; one-third selected 
books recommended by others; and one-half 
increased in their ability to explain why they 
liked or disliked a book; two-thirds increased in 
the complexity of what they wrote (they wrote 
more and gave more details about the books). 
The quotes below illustrate these changes. 

October 

"I chose this book (comic book-no title 
given) because it is a comic book. I liked the 
pictures. I hate nothing." (third-grade boy) 

May 

"I chose (Numidian Force: Quest of the 
Payland) because it had action. I liked the 
book because they were superheroes. I hated 
it because two characters had different 
clothes." (third-grade boy) 

October 

"I chose this book (New at the Zoo) because 
it is a matching book. I liked it because it was 
funny. I hated it because it was for babies." 
(fourth-grade boy) 

May 

"I chose (Numidian Force, The Isle of 
Trampling) because its a comic book. I liked 
it because the book made sense and it was 
exciting. I hated the fact that we got mixed 
up on who was the karate guy." (fourth- 
grade boy) 

There was a lack of correspondence between 
school standardized test scores and the ratings 
that the director made of participants' 
proficiency in reading. For example, of the 14 
participants who had both scores, four 
participants were beginning readers but two of 
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them scored above grade level in their 
standardized tests and two scored below. Twelve 
participants were rated as fluent readers in the 
program assessment; seven scored above grade 

level and five below on standardized tests. 

I 

• There was little change in standardized test 
scores across a year testing cycle. Of the 25 
participants for whom both reading and 
math scores were available, 52 percent were 
| above grade level in reading and 56 percent 

in math at the beginning of the year; for the 
21 participants for whom scores were 
available at the end of the year, the pattern 
remained the same with regard to being 
above or below grade level; however, four 
• participants improved slightly in reading and 

five participants improved slightly in math. 



Interfaith Neighbors 



Interfaith pursued two issues in its assessment of 
the Challenge Program. The first issue focused on the 
kinds of challenges participants undertook that were 
• related to literacy development (for example, keeping 

a journal, visiting the library regularly), social 
development (relationships with adults and peers, 
conflict management, working with others in a 
group), and leadership development (leading 
^ discussions, expressing one's opinion in a group). 

The second issue was proposed by the executive 
director, who was interested in how case managers 
used their case-management sessions. Her concern 
was that the sessions be used primarily to help young 
people address issues that emerged in the program 
) itself— their attendance, participation in activities, 

relationships with others— rather than as 
psychotherapeutic sessions that dealt with family and 
personal issues outside the program. 

» 



— 

Interfaith Neighbors 
Assessment Questions 

• What are the types of challenges (e.g., literacy- 
related, academic, social, physical, leadership) 
that youth take on in the program? 

• How do the challenges change over time? 

• How does the program support participants' 
undertaking challenges? 

• Do agency case managers make interventions 
focused on enabling the child to adapt to the 
program and to take advantage fully of group 
learning activities? 



Assessment Design 

In order to explore the questions relating to the 
first issue, the staff designed and pilot-tested pre- and 
post-surveys regarding challenges undertaken by 
participants. Separate surveys were developed for 
participants, staff and parents. Staff and parents were 
asked to rate each participant on the same issues. The 
surveys were pilot-tested during the summer 
Challenge Program, a six-week summer camp. Based 
on the responses to the questions, the surveys were 
revised and administered as pre- and post-surveys at 
the beginning and end of the 1 994-95 program year. 
AED analyzed the surveys to document change over 
time: that is, each individual was compared to 
her/himself at the beginning of the program; change 
was measured by the percentage of participants who 
went in positive or negative directions or stayed the 
same. Questions were both close and open-ended; 
open-ended questions were coded and analyzed for 
change over time. 

The second issue was also addressed through a 
survey, which the executive director developed with 
AED staff. She asked her staff to complete the survey 
three times during the year and used the surveys as 
the basis for discussions about case-management 
sessions. Findings from this part of the assessment 
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project were used internally for staff development 
purposes. 

Findings 

• Social development: The major findings from the 
surveys were that the program had the greatest 
impact on participants' social skills— participants, 
parents and staff concurred in this perception. 

► Participants: The area of greatest change that 
participants identified (40 percent saw this as 
an area of change) was that they were able to 
express their opinion in a group even if it 
went against the opinion of the majority. 

► Staff: Staff concurred in this perspective and 
added additional skills, including the ability 
to resolve problems with other participants, 
to talk about oneself in a group, and to 
express a point of view to an adult. 

► Parents: Parents concurred that the greatest 
change in their children occurred in social 
and communication skills; they were better 
able to resolve problems, listen to others, 
work in groups, and take into account 
others' perspective. They felt that their 
children were more mature, confident, and 
outgoing. 

• Literacy development: The surveys also revealed 
changes in literacy practices and skills although 
the percentages of participants who changed 
positively in these areas were lower than the 
percentages who made the social changes noted 
above. 

► Participants: 52 percent of participants felt 
that the program had some or great impact in 
helping them improve their reading skills; 48 
percent felt similarly about their writing. In 
reading, 25 percent of participants indicated 
that they were reading more short stories, 
advertisements and magazines. Smaller 
percentages indicated many other types of 



reading that they had increased; 14 percent 
indicated that they liked to read and write 
more than before they entered the program. 

► Staff: Staff rated participants' literacy changes 

as lower: they said 26 percent of participants • 

improved in their writing and 5 percent 
improved their reading. They also said that 
32 percent improved their spelling; 19 
percent improved in their ability to look up 
materials in reference books; 10 percent 4 

were more likely to read on their own. 

► Parents: Parents concurred with staff that the 
greatest changes in their children occurred in 
their social development, but 73 percent 

noted areas of academic improvement 4 

(writing, reading, spelling, critical thinking 
skills, greater focus, better grades in school). 



Pius XII 4 



Pius staff came to the AED workshops with a 
clear sense of what they wanted to evaluate. They felt 
that since their participants were recommended to 
the program because they were behind in school, 41 

they came to the program with negative images of 
themselves as learners and with negative feelings 
about the program itself. Staff wanted to assess 
whether or not their program could change these 
negative perceptions. Thus they asked whether the 
program could affect children's perception of * 

themselves as competent learners, decision makers 
and active members of a community. As staff 
discussed these ideas in the workshops and with 
AED staff, they listed indicators of change in these 
areas. Increasingly, they came to believe that their 4 

assessment goal was too ambitious and that the most 
important area for them to investigate was decision 
making because it was an area that they could 
influence and measure. This was also an area that fit 
with the program goal of helping young people 
become "individuals who take action" rather than 4 

"individuals to whom things are done." 
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Pius XII 

Assessment Questions 

• How does the tutoring program encourage 
decision making in youth? 

• What training and support are available for staff 
with regard to encouraging decision making in 
youth? 

• What changes are observed in participants over 
time? 

• Are program behaviors (decision making) 
carried over to school? 



Assessment design 

More than any other program, Pius debated the 
questions for the evaluation and also used the 
process of developing the design as a staff 
development opportunity. For example, in focusing 
the evaluation on decision making, staff came up 
with a comprehensive list of 19 areas in which 
participants could make decisions. Developing the 
list increased the consciousness of staff about 
decision-making opportunities. As a result of the 
discussions, the director said, "Decision making was 
constantly under discussion— it worked its way into 
people's thinking and they could articulate this." 
Staff also developed instruments that increased the 
amount of reflection that went on among staff and 
participants about decisions. These instruments 
included dialogue journals that staff maintained and 
shared with the director; dialogue journals 
maintained by tutors and shared with group leaders; 
and monthly evaluation forms kept by participants 
about what they had learned and what they wanted 
to learn; and participant reading logs. Because of the 
impact of budget cuts on the program, the program 
lost its tutors. These were replaced by volunteers 
who had to be trained; because of these new 
responsibilities, the director was backlogged in 
responding to staff journals and was forced to 



discontinue them. However the participants' reading 
logs and monthly evaluation forms were maintained. 
In addition to these evaluation activities, Pius XII also 
developed and administered a parent survey to find 
out the kinds of literacy activities participants 
engaged in at home and the kinds of literacy 
activities that parents engaged in. 

Findings 

• Literacy development: 25 participants' reading 
logs and monthly evaluation forms were 
analyzed. 

► They revealed that participants completed 
goals in reading (they set targets for 
themselves in reading at the beginning of the 
year which they were able to accomplish). 

► Their logs revealed that they were writing 
more. 

► Three-fourths of the participants who wrote 
nothing at the beginning of the year had 
entries in March regarding their strengths 
and what they hoped to improve. 

► The focus of participant writing changed 
over time. In the beginning they wrote very 
generally about their likes and dislikes, but 
by March their writing reflected specific 
interests that they developed in the program 
through working on the ecology project, 
What Survives in the Bronx. 

• Parents' role in literacy development: The 

survey helped to dispel the myth that parents of 
poor children do not provide and support 
opportunities to read. 

► The parent survey revealed that most of the 
30 responding parents said their children 
read at home but the reading was primarily 
school-related (worksheets and textbooks). 
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► Their children had a special place to read, 
but not a fixed time. 

► Their children asked for help with their 
homework if they needed it. 

► Most parents read to their children although 
very few read daily. 

► Parents themselves liked to read and had 
many other literacy-related interests. 

► The survey also provided information that 
could be used to promote other literacy 
practices at home— for example, reading for 
pleasure at regular times. 



Project Reach Youth (PRY) 



PRY staff focused on two areas of its Family 
Learning Center program: the impact of homework 
time on participants' ability to do homework and on 
their school performance, and the impact of the 
thematic units on participants' knowledge and skill 
development. 

The first theme resulted from the staff's 
ambivalence about the role of homework in their 
program. They were not certain how useful the time 
spent in doing homework was to participants; what 
teachers' expectations were; and whether or not 
parents felt that the homework part of the program 
was useful. To address the latter question, early on in 
the project, the staff conducted a focus group with 
parents. The parents felt that having their children do 
homework in the program was very important. 
Parents noted that even the youngest children had 
quite a bit of homework and that their children had 
difficulty prioritizing tasks in homework. The second 
theme emerged from the directors' desire to 
document the types of instructional approaches used 
in the program and to promote staff assessments of 
the effectiveness of these approaches. 



PRY 

Assessment Questions 

How does the program affect participants' 
ability to do homework? Do participants 
become more independent in doing 
homework? 

How effective is the program in teacliing 
participants through thematic units? 



Assessment Design . 

Based on the parent focus group, PRY staff 
designed a log that participants completed every day 
over a four-week period regarding the subjects in 
which they had homework, the assignments they 
had, which ones they completed, and what they 
chose to do in choice time. The staff also designed a 
survey that they conducted with 10 teachers — five 
each from the two participating elementary 
schools— regarding their expectations in assigning 
homework (how much was assigned, in what 
subjects, for what purposes); why participants failed 
to complete homework; and how effective they felt 
homework was for participants' learning. Staff also 
wanted to know if improvement in homework grades 
was correlated with improvement in academic 
grades. To carry out this aspect of the study, they 
collected grades for participants from two marking 
periods. 

The second part of the assessment focused on the 
impact of thematic teaching on participants' learning. 
Program staff were asked to complete Instructional 
Record Forms in which they described what they 
were teaching during a two-month unit on city life. 
They described the activities they carried out, the 
instructional approaches they used, what went well 
and what difficulties they had, and what they noticed 
about participant learning during the unit. 



4 



4 



4 



4 



4 



4 



4 



I 



4 



o"~i 

ERIC 



A :o. ; Act A ■ © i*!;' E d u o at I on a l b e ■ ,v Alllo pm . i»-i : h A- 



« 



PARTICIPATORY ASSESSMENT IN AFTERSCHOOL PROGRAMS 



Summary Report 



Page SB 



Findings 

• Classroom teachers' views of homework: Staff 
found that the 10 teachers thought homework 
was important and effective in improving 
participants' learning. In both schools, teachers 
had very similar expectations regarding 
homework — the amount they assigned, the 
number of subjects, the purposes of homework, 
and the reasons they gave for why participants 
did not do their homework. Teachers gave 
virtually the same ratings to the following 
purposes for homework: review, foster positive 
work habits, and foster independent learning. 
They also claimed that 75 percent of participants 
completed homework; the main reasons for not 
completing homework were participants' home 
situations, and student laziness and lack of self- 
discipline. 

• Participants' self-report on homework: Many 
assignments described by participants 
emphasized rote learning— completing a 
worksheet or writing a list of spelling words three 
times. Very seldom were participants asked to do 
homework that demanded thought beyond what 
had been already covered in class. Most 
participants completed their math homework in 
the program. Even when other subjects were 
assigned, this was the one area chosen most 
frequently. 

• Relationship between homework grades and 
grades in academic subjects: 42 percent of 
participants in the program improved their 
school homework grades; 46 percent stayed the 
same; and 12 percent decreased. Of those who 
improved, 83 percent also improved their grades 
in core subjects; of those who stayed the same, 
65 percent improved in core subjects; of those 
who decreased, 62 percent increased their 
grades in core subjects. 

• Learning in thematic units: Teachers revealed 
that the thematic units were multidisciplinary, 
involving math, reading, and writing skills, and 



focusing on geography, multicultural, and social 
issues. For example, among the activities in the 
thematic unit were: 

► Learning about city workers 

► Practicing math skills and using word 
problems that involved city life 

► Writing poems about the city and 
developing monologues of city life 

► Writing and performing skits about city life 

► Identifying the five boroughs on a map and 
favorite places participants had visited 

► Learning about cultural diversity by sampling 
various ethnic foods 

► Listening to urban stories of various ethnic 
groups and reading about these groups 

► Discussing city problems and ways to solve 
them 

• Skill Development: Teachers identified the 
following areas of participant development as a 
result of these activities: 

► Improved ability to work in groups 

► Greater comfort with writing 

► Improved ability to present work to others 
through various media 

► Learning how to comment on the work of 
other participants 

Lessons Learned 

In a review of the main lessons learned from the 
evaluation, program staff spoke enthusiastically about 
the process as well as about the findings. Their 
remarks emphasized the following lessons: 
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1. The assessment findings validated many 
aspects of what the programs were doing. The 
findings provided evidence that young people were 
indeed reading and writing more and reading more 
complex material over the course of the program and 
that they were making decisions, undertaking 
challenges and accomplishing something in the 
periods of time devoted to homework and choice 
time. This validation gave staff courage to face what 
one described as the "gorilla on our backs"— namely 
the expectation of their boards and foundations that 
their work would result in improvements in 
participants' standardized test scores or grades. The 
programs felt that they needed to have, and indeed 
were developing, alternative ways to measure and 
demonstrate participant learning. 

2. The assessment process helped programs to 
clarify their objectives and think about whether 
their program activities supported these objectives. 
Certainly the discussions about choice and decision 
making helped program staff clarify conceptually 
complex outcomes— such as empowerment— and be 
more specific about what types of behaviors and 
attitudes they wanted to promote in their programs 
that would foster these outcomes. 

The assessment process gave staff a common 
language to talk about key aspects of their programs. 
For example, in developing surveys for the Challenge 
Program, Interfaith staff had to clarify what they 
meant by challenges, what type of challenges existed 
in their programs, and how they supported children 
in meeting these challenges. Similarly, the 
discussions around homework raised interesting 
questions for PRY staff about the purposes of 
homework and the possibilities of dialogue between 
classroom and afterschool teachers about homework 
and its relationship to learning. 

3. The assessment project opened new ways of 
thinking about and documenting outcomes— for 
example, documenting proficiency in reading and 
writing through student logs. The assessment 
process helped staff become proficient in developing 
new forms to document important outcomes. These 
forms included reading logs, monthly evaluations, 



€ 

homework logs, staff journals, pre- and post-tests of 
reading, writing and math assessment instruments, 
and parent surveys. These forms will be retained and 
used in the future to assist in program planning and 
development.' 

* 

4. The assessment provided insight into what 
program staff needed to focus on in planning 
program improvements and helped them act in 
response to the findings. Several programs modified 

their schedules and made programmatic decisions ^ 

while in the process of conducting the evaluation. 

One program changed the schedule to provide more 
time for community building; another concluded that 
45 minutes was sufficient for homework; another that 
participants needed more opportunities for choice. 

Staff in one program decided that their focus was still 1 

too rooted in a social work perspective focusing on 
psychosocial problems and that they needed to shift 
to a focus on the young person as learner and to 
continue to reorient their case management even 
more than they had to this perspective. ^ 

5. What programs learned about the assessment 

process made them more knowledgeable in 
discussions with board members, foundations, and 
other funders about outcomes and how to 
document them. 4 

6. Staff realized that in order to have the kinds 

of important discussions required for an assessment, 
they needed time and resources. As a result, they felt 
it was important to build time into future proposals to 
foundations, including time for part-time staff to * 

participate in the assessment process. 



i 



* The development and use of the instruments helped 
staff enhance and rely on their observational skills and 
translate observations into systematically organized data 
about outcomes — i.e., the programs moved from recording 
anecdotes to reporting results. 
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Chapter Four 

Recommendations for Participatory 
Assessment Projects and for Developing 
Effective Afterschool Programs 

Recommendations 

I n assessing the effectiveness of the participatory 
assessment project, AED used the following 
indicators of effectiveness: 

• Continuity in staff participation and 
commitment to carrying out the assessment 
tasks 

• Commitment to using the findings from the 
assessment in improving the program 

• Institutionalization of approaches to assessment 
as evidenced by use of documentation 
instruments and involvement of other staff in 
the assessment process 

• Using what was learned in communicating 
with program constituencies 

All the programs in the assessment project made 
the kinds of commitments and followed through in the 
ways described above. AED believes the following five 
aspects of the project contributed to its effectiveness: 
program selection; ownership of the assessment 
process; networking among programs; technical 
assistance; and the multi-stage process. These are 
described briefly below. 

/. Program selection. Careful program selection 
is very important to ensure that programs have the 
capacity to carry out the project. Through the selection 
process, AED was able to assess program commitment 
to the project and capacity to carry it out; programs in 
turn were able to assess whether or not the time that 
the project demanded would be worthwhile. The 
criteria that AED used in program selection included: 

• Commitment of staff to participate in all 
phases of the project 



• Participation of staff who are in leadership 
positions in which they would be able to apply 
what they learned in the project in their 
programs 

• Sufficient organizational stability so that 
staff could participate in a two-year project, 
carry out assessment tasks in addition to their 
normal work load, and share their learning 
with others 

• Already existing organizational support as 
well as structures for staff development and 
inquiry that could be used to discuss 
assessment activities and findings 

2. Ownership of the assessment process. It is 
essential that programs develop ownership of the 
assessment process through developing questions that 
are important to them; developing their own designs 
that reflect program objectives and activities; carrying 
out the data collection and analysis when feasible; and 
reviewing and critiquing any materials written on their 
programs. Once programs participating in the 
assessment project learned something about the 
purposes and types of evaluation and about the kinds of 
questions that could be addressed, they developed their 
own questions. This process involved reporting on what 
they had learned in workshops to other program staff; 
reviewing their objectives; and discussing aspects of 
their programs— such as the use of time and case 
management — the effectiveness of which they 

questioned. Staff were willing to ask questions of 
consequence to their programs because they felt that 
the assessment findings would not be used against 
them— as the findings from non-participatory or 
monitoring types of evaluations often are. 

This sense of ownership was further enhanced 
when programs developed documentation forms and 
other assessment instruments that they continued to use 
after the project ended. These instruments provided 
data on program participants and parents; how staff 
carry out their jobs; the nature and intensity of 
activities that young people engage in; and the impact 
of various literacy activities on young people's skills. 
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3. Networking among programs. This occurred 
through the workshops which provided participants 
with valuable information gained from listening to their 
colleagues and validated their efforts by demonstrating 
commonalities in approaches and obstacles. During the 
course of the project, networking became one of the 
most important factors in sustaining program interest in 
the assessment. Although at first reluctant to share 
beyond the level of "show and tell," after a relatively 
short period of time, program staff were able to trust 
their colleagues and to share program strategies, 
problems, and assessment instruments freely. This was 
remarkable since the project took place in a period of 
declining resources for afterschool programs. 

4. Technical assistance . AED's role in the project 
was critical both in terms of providing technical 
assistance to programs in the areas in which they 
lacked expertise: developing and refining assessment 
questions; developing documentation forms; and 
collecting, entering and summarizing data. AED also 
affirmed the collaborative nature of the project by 
modeling collaborative, participatory methods in the 
workshops where everyone was both learner and 
expert; by respecting programs' priorities— for 
example, during the budget crises, understanding the 
priority of serving young people rather than 
participating in the project; and by viewing program 
input into all phases of the project as essential. 

5. Multi-stage process. Having several distinct 
phases in the project was important to maintaining 
momentum and sustaining interest. The phases 
included: 

• Learning about evaluation and evaluation 
strategies 

• Developing questions and an assessment design 

• Communicating the design to other programs, 
which provided important validation for the 
programs and also provided a deadline for 
completion of this phase of the project 

• Data collection and analysis 



• Participating in analyzing findings and 
reviewing written materials on the project 

• Reflecting on findings and making adjustments 
in response 



Esential Elements of Youth-Serving Programs with 
a Focus on Literacy 

As indicated throughout this report, the afterschool 
programs that participated in the assessment project 
were strong programs. In many ways, these programs 
are models of what good youth-serving programs 
should be. The following section discusses eight 
important elements that should characterize effective 
youth-serving programs: a philosophy of literacy and 
youth development; effective leadership; staff 
commitment; an "un-school-like" environment; a sense 
of community; ongoing staff development; the 
involvement of parents; and effective relationships with 
the schools. (Seven of these elements characterized the 
afterschool programs that participated in the assessment 
project to some degree; the eighth did not although it 
would certainly enhance their effectiveness— and that 
of youth-serving programs in general.) 

1. A philosophy of literacy and youth 
development. To be effective programs need a 
philosophy of literacy and youth development that 
provides a framework and guide for developing 
activities. All the programs espoused a philosophy of 
youth and literacy development and a pedagogy that 
emphasized teaching for meaning and active 
participation of the learner in constructing meaning. 
Although the specific activities in each program were 
different, there were broad similarities in philosophy. 
Central to their perspective was a view of young people 
as resources to be developed, not as problems to be 
fixed. This perspective informed the programs' views 
of, and created an emphasis on, learner strengths and 
on providing possibilities for learning arising from 
what participants already knew and wanted to do. 

Having a strong philosophical orientation is 
essential in afterschool programs because, without it, 
there is no basis for staff or program development. 
Without a philosophical basis, a program is in danger 
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of becoming a baby-sitting service in which individual 
staff members develop activities reflecting their own 
experiences or background rather than a coherent 
program grounded in best practices. 

2. Effective leadership. Programs require effective 
leadership that espouses and articulates the guiding 
philosophy and principles; that provides opportunities 
for staff development, including ongoing participation 
in program planning and development and a 
supportive, collaborative atmosphere in which to work; 
and that is responsive to the parents of participants. 
Additional leadership skills include accessing resources 
and collaborating with other agencies and community 
members in strengthening communities. 

3. Staff commitment. Programs require staff who 
are committed to young people, their families and the 
communities from which they come. Staff in 
afterschool programs are often paid low salaries and 
therefore their greatest rewards come from the 
knowledge and satisfaction that they are able to 
positively influence young people's development. They 
must have respect for and knowledge of young people's 
families and communities in order to understand their 
lives and to build on their strengths. 

4. An un-school-like environment. Programs must 
consciously develop environments that are not school- 
like in order to provide young people with alternative 
routes to learning without the negative connotations that 
many participants associate with school. All the 
programs recognized the need to provide learning 
activities that provided opportunities for young people 
to develop their minds in many directions and to 
experience themselves as competent, active learners. 
These activities, while often not directly related to 
improved school performance, developed young 
people's sense of themselves, helped them find their 
"voice," and strengthened their ability to articulate 
their knowledge and opinions in writing and other 
formats. 

5. A sense of community. Programs must 
consciously build a community based on respect, trust 
and support. The programs' ideas for community 
building came from their understanding of literacy as 



well as from a recognition of the importance of trust 
and group support in youth development. They 
promoted literacy development by having students 
share their ideas, books, writing, and art work with 
their peers as well as the larger community. They 
emphasized the idea that people communicate best 
when they trust the people to whom they are 
communicating and when they see that what they 
communicate is valued. The programs also promoted 
youth leadership through encouraging choice, decision 
making, and young people's participation in the 
development and maintenance of group norms of 
respect and support. The sense of community that 
existed within programs was enhanced by the 
collaboration among staff involved in assessing 
participant needs and progress. 

6. Ongoing staff development. Staff development 
in both youth development and literacy must be 
ongoing. Not all the programs hired staff of 
comparable sophistication or experience. For example, 
one program used college tutors, while others hired 
experienced teachers, social workers and youth 
workers. In spite of these differences in staff 
backgrounds, all the programs were committed to 
having staff engage in ongoing learning experiences 
through program planning, reviewing the development 
of individual participants, and, when possible, 
providing opportunities for staff to learn new skills. 
The project itself deepened the discourse by demanding 
that staff scrutinize their own practices more closely 
and by providing them with opportunities to assess 
whether or not program activities reflected program 
goals and objectives. 

7. Parent involvement. Programs need to consult 
parents about their children's development as well as 
about the impact of the program on them. All the 
programs involved parents in assessing the program as 
well as in discussions of their children's development. 
The project increased the involvement of parents in 
program assessment by conducting parent surveys and 
focus groups, through which programs learned how 
parents assessed the programs' impact on their 
children's literacy and social development; what their 
expectations were of the program— for example the 
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importance attached to doing homework; and how 
parents perceived the literacy activities that their 
children engaged in at home. The project developed the 
capacity and interest of programs in further involving 
parents in the assessment of their programs. 

8. Relationships with schools. Programs and 
schools need to develop relationships with one another 
focusing on addressing the developmental needs of the 
young people. All but one program in the assessment 
project had some relationship with the sending schools. 
Nevertheless, none of the relationships was ideal in that 
none of them focused on developing a holistic 
perspective on the learning and developmental needs of 
the young people in the program. This was not the fault 
of either the programs or the schools, neither of which 
had been asked to develop collaborative relationships or 
given the resources to do so. One program attempted 
to bridge the gap between program and school through 
its study of homework and interviews with teachers 



about their expectatioas. In general, the relationships 
between the atterschool programs and schools were 
characterized by a lack of communication and 
sometimes hostility. Program staff felt that often 
teachers did not serve young people well with the result 
that young people came to them damaged by their 
experiences in school; and some teachers lacked 
respect for the services provided by atterschool 
programs, seeing them primarily as baby-sitting 
services or homework helpers. This communication 
gap needs to be bridged so that a holistic perspective on 
young people and mutual respect for the vital role that 
both schools and community-based, youth-serving 
programs play in the education of youth can develop. 
While requiring additional time and resources, 
improved communications between school and 
community programs would help each meet the needs 
of youth more effectively. 
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